
The Office Action objects to the claims on the ground that originally numbered 
claims 25-34 are misnumbered, there being no original claim 24. The Office Action 
renumbered claims 25-34 as claims 24-33. This renumbering is adopted by this 
Amendment and renumbered claims 25-32 have been amended to conform their 
dependencies to the renumbering. 

Claims 27, 28 and 32 have also been amended to correct an antecedent issue by 
changing "audio/visual" to — audio/video — . 

The Office Action rejects claims 1, 2, 4, 5, 7 and 10 under 35 U.S.C 102(e) as 
anticipated by U.S. Patent No. 6,272,457 to Ford et al, hereafter Ford. Ford does not 
teach or disclose the examining step, the automatically generating step or the analyzing 
step recited in claim 1 . 

The Office Action cites the passage at column 2, lines 34-36, as support for the 
examining step. However, this passage merely refers to editing tools that a field user can 
use to review and correct speech entries. There is no mention of examining the 
information stream to recognize a presence of events that occur in the information stream 
as recited in claim 1 . 

The Office Action cites the passage at column 2, lines 31-47, as support for the 
automatically generating step. However, this passage merely refers to building a data 
record of corrected location data and recognized text data synchronized to a location or a 
track, the data record being stored in a geographic information system (GIS) database and 
correlated to linear networks and point observations. There is no mention of the step of 
automatically generating database queries from the recognized events of the examining 
step as recited in claim 1 . 

The Office Action cites the passage at column 4, lines 47-49, as support for the 
analyzing step. However, this passage merely refers to analysis/processing of asset 
information developed during collection. There is no mention of analyzing database 
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query results so as to rank and select database query results to be inserted into the 
information stream as collateral information as recited in claim 1 . 

With respect to claim 2, the Office Action cites the passage at column 6, 
beginning at line 61 and ending at column 7, line 2. This passage refers to a correlation 
of the asset information to reference data. There is no mention of an analyzing step 
comprising a step of ranking the database query results based on a plurality of criteria as 
recited in claim 2. 

With respect to claim 4, the Office Action cites the passages at column 5, lines 
39-45 and at column 8, lines 7-9. Neither of these passages mentions a topic taxonomy 
used in automatic text classification as recited in claim 4. 

With respect to claim 5, the Office Action cites the passage at column 8, lines 47- 
52. This passage does not mention a step of automatically extracting text from the 
information stream as recited in claim 5. 

With respect to claim 7, the Office Action cites the passage at column 5, lines 8 
and 9. This passage does not mention a step of automatically extracting text from the 
information stream that further includes a step of operating a voice recognition system, as 
recited in claim 7 

With respect to claim 10, the Office Action cites the passage at column 5, lines 3- 
7. This passage refers to the use of photos but does not mention a step of also generating 
text that is descriptive of a number of human faces present in an image conveyed by the 
information stream as recited in claim 10. 

For the reason set forth above, it is submitted that the rejection of claims under 
35 LLS.C. 102(e) as anticipated by Ford is erroneous and should be withdrawn. 
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The Office Action rejects claims 3, 6, 8, 9 and 1 1 under 35 U.S.C 103(a) as 
unpatentable over Ford in view of an article entitled "Topic Detection and Tracking Pilot 
Study Final Report", James Allan et al., Proceedings of the DARPA Broadcast News 
Transcription and Understanding Workshop, pages 194-218, 1998, hereafter Allan. 

Ford does not teach the steps of base claim 1 of claims 3, 6, 8, 9 and 1 1 or any 
intervening claims as noted above in the discussion concerning the rejection of base 
claim 1 and the intervening claims under 35 U. S. C. 102(e). Allan was cited as teaching 
the features recited in claims 3, 6, 8, 9 and 1 1, but not as teaching the steps of base claim 
1 and the intervening claims. Accordingly, neither Ford nor Allan teaches the subject 
matter of base claim 1 or of the intervening claims. Therefore, the conclusion of 
obviousness is erroneous. 

The conclusion of obviousness is also erroneous because Allan does not teach the 
features recited by claims 3, 6, 8, 9 and 11. 

With respect to claim 3, Allan's HMM model process does not teach searching a 
collateral information source or the use of a topic taxonomy and text categorization or the 
use of the topic taxonomy to compare the similarity of collateral information queries and 
their search results and to rerank and select collateral information search results. 

With respect to claim 6, Allan does not teach the step of operating on the 
sentences to identify topics that correspond to predetermined topic taxonomies and 
wherein the step of automatically generating database queries operates on the identified 
topics. 

With respect to claim 8, Allan does not teach the step of extracting closed caption 

text. 

With respect to claim 9, Allan does not teach the step of operating a character 
recognition system. 
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With respect to claim 11, Allan describes the "Dragon approach of using an 
HMM to solve the very specific problem of segmenting the data stream into individual 
news story topics". This does not address the problem of automatically generating 
collateral information queries and reranking and selecting collateral information search 
results. 

For the reasons set forth above, it is submitted that the rejection of claims 3, 6, 8, 
9 and 1 1 under 35 U.S.C 103(a) is erroneous and should be withdrawn. 

The Office Action rejects claims 12 and 15-17 under 35 U.S.C 103(a) as 
unpatentable over Ford in view of U.S Patent No. 5,970,460 to Bunce et al, hereafter 
Bunce. 

Ford does not teach the step of analyzing the text to identify information elements 
as recited in claim 12. The Office Action cites the passage at column 2, lines 29-41 , as 
showing this step. However, this passage merely refers to editing tools that a field user 
can use to review and correct speech entries. There is no mention of examining the text 
to identify information elements as recited in claim 12. Bunce does not teach this step 
either. Since neither Ford nor Bunce teach the analyzing step, the conclusion of 
obviousness is erroneous. 

The Office Action concedes that Ford does not teach the steps of automatically 
generating, extracting and multiplexing as recited in claim 12, but alleges that Bunce 
does teach these steps. This allegation is erroneous with respect to the automatically 
generating, extracting and multiplexing steps. 

In support of the automatically generating step, the Office Action refers to the 
passage at column 6, lines 48-52 of Bunce. However, this passage merely refers to 
accessing the database that contains the text that is being edited to match edited text with 
original test. The passage makes no mention of automatically generating queries from 



7 



the information elements of the analyzing step to search the database as recited in claim 
12. 

In support of the extracting step, the Office Action refers to the passage at 
column 6, lines 61-65 of Bunce. However, this passage merely refers to user selection of 
a synthesized entry, which is not changed or extracted in any way, but remains 
unchanged by the user selection. The passage makes no mention of extracting data from 
the database search results that is relevant to the information stream as recited in claim 
12. 

In support of the multiplexing step, the Office Action refers to the passage at 
column 7, lines 52-60, of Bunce. However, this passage merely refers to presenting the 
user with options when the user selects a word in text by cursor operation. The passage 
makes no mention of multiplexing the data from the database search results into the 
information stream for presentation at a destination of the information stream as recited 
in claim 12. 

Bunce merely describes a system for editing and correcting transcripts generated 
by an automatic speech recognition system. Bunce's only relevance to Ford or to the 
invention recited in claim 12 is the step of converting and editing the information stream 
to text. The Office Action's reading of the automatically generating and extracting steps 
is strained and unreasonable, as Bunce relates solely to speech recognition and editing 
thereof and not to augmenting the converted text with relevant collateral information as 
claimed in claim 12. 

With respect to claims 15-17, the cited passages do not mention generating the 
queries based on information elements that correspond to a list of information elements 
identifying topics in the text being analyzed, where the topics correspond to 
predetermined topic taxonomies. The passages make no mention of topics or topic 
taxonomies. 
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The Office Action suggestion to use the relied upon teaching of Bunce in 
combination with Ford is improperly based on the hindsight of Applicants' disclosure. 
Such hindsight reconstruction of the art cannot be the basis of a rejection under 35 U.S.C. 
103. The prior art itself must suggest that modification or provide the reason or 
motivation for making such modification. In re Laskowski , 871 F.2d 115, 117, 10 USPQ 
2d 1397, 1398-1399 (CAFC, 1989). "The invention must be viewed not after the 
blueprint has been drawn by the inventor, but as it would have been perceived in the state 
of the art that existed at the time the invention was made." Sensonics Inc. v. Aerosonic 
Corp . 38 USPQ 2d 1551, 1554 (CAFC, 1996), citing Interconnect Planning Corp. v. Feih 
774 F. 2d 1132, 1138, 227 USPQ 543, 547 (CAFC, 1985). 

For the reasons set forth above, it is submitted that the rejection of claims 12 and 
15-17 under 35 US.C. 103(a) is erroneous and should be withdrawn. 

The Office Action rejects claims 13 and 14 under 35 U.S.C 103(a) as 
unpatentable over Ford in view of Allan and further in view of Bunce. 

This rejection is erroneous for the same reasons set forth above in the discussion 
concerning the rejection of base claim 12 under 35 U.S.C. 103(a). 

The Office Action concedes that neither Ford nor Bunce teaches the step of 
ranking as recited in claims 13 and 14, but alleges that Allan does. Allan describes a 
technique for clustering news stories using Group Average based Clustering (GAC). The 
clusters, in turn, are used to track and link together multiple stories (and events) about the 
same news topic. This teaches nothing about how to rank and select relevant items of 
information from a collateral information search result using a plurality of criteria (e.g., 
search relevance score, number of overlapping named entities, categorization taxonomy 
path similarity, etc.). Rather, Allan merely shows how related news stories might be 
grouped together. 
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For the reasons set forth above, it is submitted that the rejection of claims 13 and 
14 under 35 US.C. 103(a) is erroneous and should be withdrawn. 

The Office Action rejects claims 18-23 under 35 U.S.C 103(a) as unpatentable 
over Bunce in view of U.S Patent No. 5,786,814 to Moran et al., hereafter Moran. 

This rejection is erroneous because the allegations in the Office Action that Bunce 
discloses a subsystem for examining and a subsystem for automatically generating as 
recited in claim 18 are erroneous. 

In support of the subsystem for examining, the Office Action cites the passage 
beginning at column 1, line 63 and ending at column 2, line 3, of Bunce. This passage 
refers to a speech event database (SEDB) in which is stored converted audio text and that 
is maintained in association with an editing program. When a user of the editing program 
initiates a change in the editing point of a document, a preselected region of text around 
the new editing point is extracted and cached to generate a context list. The passage does 
not mention any subsystem for examining an information stream to recognize a presence 
of events that occur in the information stream as recited in claim 18. 

In support of the subsystem for automatically generating, the Office Action cites 
the passage at column 2, lines 3-5, of Bunce. This passage refers to searching the SEDB 
for text that matches the context list. The passage does not mention any subsystem, 
having an input coupled to an output of the examining subsystem, for automatically 
generating database queries from the recognized events as recited in claim 18. 

Accordingly, Bunce does not disclose or teach the subsystems for examining and 
automatically generating as recited in claim 18. Moran does not teach these subsystems 
either. Therefore, the conclusion of obviousness is erroneous. 

The Office Action concedes that Bunce does not teach the subsystem for 
analyzing as recited in claim 18, but alleges that Moran does. 
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In support of the subsystem for analyzing, the Office Action cites the passage at 
column 3, lines 12-40 of Moran. This passage generally describes a user interface for 
controlling playback of temporal data representing a collaborative activity such as a 
meeting. The passage makes no mention of a subsystem, having an input coupled to an 
output of the database, for analyzing database query results so as to rank and select 
database query results to be inserted into the information stream as collateral information 
as recited in claim 18. 

Accordingly, since Moran fails to teach the analyzing subsystem of claim 18, the 
conclusion of obviousness is erroneous. 

With respect to claim 19, the cited passages of Bunce have nothing to do ranking 
criteria, a taxonomy path score, queries base on topics as recited in claim 19. 

With respect to claim 20, the cited passages of Bunce do not teach that the 
examining subsystem comprises an extracting unit, a segmenting unit and a topic 
identifying unit as recited in claim 20. 

With respect to claim 22, the cited passage of Moran merely mentions events of a 
video stream that include a person, but do not mention that an examining subsystem 
comprises a unit for generating text that is descriptive of a number of human faces that 
are present in an image conveyed by the information stream as recited in claim 22. 

With respect to claim 23, the passage at column 16, lines 7-9, of Moran merely 
states that temporal analysis is particular to the information captured by the time stream 
and the desirable events to be captured, but does not mention a unit for operating on the 
sentences to identify the names of entities as claimed in claim 23. The passage at column 
16, lines 33-42, of Moran refers to an index of the events of an object, but does not 
mention that a query generation subsystem assembles a query object for searching a 
database to identify a first set of documents that contain the named entities as claimed in 
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claim 23. The passage at column 16, lines 38 and 39, of Moran does not even mention a 
second set of documents as claimed in claim 23. The passage at column 16, lines 26-45, 
of Moran not only fails to mention the first and second documents, but also does not 
mention that an analyzing subsystem scores the returned first and second sets of 
documents and ranks the documents based on their scores as claimed in claim 23. 

For the reasons set forth above, it is submitted that the rejection of claims 18-23 
under 35 US.C. 103(a) is erroneous and should be withdrawn. 

The Office Action rejects claims 24-26 and 31-33 under 35 U.S.C 103(a) as 
unpatentable over Bunce in view of Ford. 

With respect to claim 24, the Office Action concedes that Bunce does not teach 
any of the steps of claim 24, but that Ford teaches all of the steps of claim 24 and, 
therefore, the combination Bunce and Ford renders claims 24 obvious. 

This rejection is untenable. 

Ford does not teach the examining step of claim 24*. The Office Action cites the 
passage at column 2, lines 34-36, of Ford as support for the examining step. However, 
this passage merely refers to editing tools that a field user can use to review and correct 
speech entries. There is no mention of examining the information stream to recognize a 
presence of events that occur in the information stream as recited in claim 24. 

The Office Action cites the passage at column 2, lines 31-47, of Ford as support 
for the generating step of claim 24. However, this passage merely refers to building a 
data record of corrected location data and recognized text data synchronized to a location 
or a track, the data record being stored in a geographic information system (GIS) 
database and correlated to linear networks and point observations. There is no mention 
of the step of generating database queries from the recognized events of the examining 
step as recited in claim 24. 
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The Office Action cites the passage at column 4, lines 47-49, of Ford as support 
for the analyzing step of claim 24. However, this passage merely refers to 
analysis/processing of asset information developed during collection. There is no 
mention of analyzing database query results so as to rank and select database query 
results to presented as the collateral information as recited in claim 24. 

For the reasons set forth above, Ford does not teach the examining, generating 
and analyzing steps or instructions of claim 24. Therefore, the conclusion of obviousness 
is erroneous. 

With respect to claim 25, since Ford does not does not teach either an audio/video 
information stream or the steps of examining, generating and analyzing as discussed 
above, Ford cannot teach inserting the output of such steps, i.e., the collateral 
information, into the audio/video information stream. 

With respect to claims 26, 31 and 32, the Office Action contradicts the basic 
concession of the rejection, namely, that Bunce does not teach the steps recited in claims 
24. 

With respect to claim 26, the cited passage of Bunce merely deals with speech 
recognition and makes no mention that the recognized events comprise speech topics as 
claimed in claim 26. 

With respect to claim 3 1, the cited passages of Bunce have nothing to do with 
generating database queries based on information corresponding to identified topics 
extracted from the audio/video stream, where the topics correspond to predetermined 
taxonomies as recited in claim 3 1 . 

With respect to claim 32, the passages at column 5, lines 44-67, and at column 6, 
lines 11-18 and 48-52, of Bunce have nothing to do with identifying topics in the 
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sentences that correspond to predetermined topic taxonomies or with the generating step 
operating on the identified topics as claimed in claim 32. 

With respect to claim 33, The Office Action alleges that Bunce teaches the 
examining instruction, citing the passage at column 3, lines 15-23. This passage merely 
describes a threshold for distinguishing the spoken words from background noise and has 
nothing to do with examining the information stream to recognize a presence of events 
therein as recited in claim 33. 

For the reasons set forth above, it is submitted that the rejection of claims 24-26 
and 31-33 under 35 US.C. 103(a) is erroneous and should be withdrawn. 

The Office Action rejects claim 27 under 35 U.S.C 103(a) as unpatentable over 
Bunce in view of Ford and further in view of Allan. 

This rejection is erroneous for the reasons set forth in the discussion concerning 
the rejection of base claim 24. Thus, it is submitted that the rejection of claim 27 is 
erroneous and should be withdrawn. 

The Office Action rejects claims 28-30 under 35 U.S.C 103(a) as unpatentable 
over Bunce in view of Ford and further in view of Moran. 

This rejection is erroneous for the reasons set forth in the discussion concerning 
the rejection of base claim 24. Thus, it is submitted that the rejection of claims 28-30 is 
erroneous and should be withdrawn. 

Attached hereto is a marked-up version of the changes made to the specification 
and claims by the present amendment. The attachment is captioned "Version With 
Markings To Show Changes Made." 
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It is respectfully requested for the reasons set forth above that the rejections under 
35 U.S.C. 102(e) and 35 U.S.C. 103(a) be withdrawn, that claims 1-33 be allowed and 
that this application be passed to issue. 



Respectfully Submitted, 



Date: l-WOI "ftj^ | 

Paul D. Greeley 




Reg. No. 31,019 
Attorney for Applicant(s) 
Ohlandt, Greeley, Ruggiero & Perle, L.L.P. 
One Landmark Square, 10 th Floor 
Stamford, CT 06901-2682 
(203) 327-4500 
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VERSION WITH MARKINGS TO SHOW CHANGES MADE 



Application, Serial No. 09/698,894 

IN THE CLAIMS 

Please amend the claims as follows: 

25. (Amended) A method as in claim [25] 24, and further comprising a step of 
inserting the collateral information into the audio/video stream in real time or 
substantially real time. 

26. (Amended) A method as in claim [25] 24, wherein the step of examining includes 
a step of generating a speech transcript from at least the audio portion of the audio/visual 
stream, and wherein recognized events comprise speech topics. 

27. (Amended) A method as in claim [25] 24, wherein the [audio/visual] audio/video 
stream originates as a television broadcast signal. 

28. (Amended) A method as in claim [25] 24, wherein the [audio/visual] audio/video 
stream originates at a meeting, and further comprising a step of presenting the collateral 
information to meeting participants in real time or substantially real time. 

29. (Amended) A method as in claim [29] 28, wherein the step of presenting 
comprises a step of inserting the collateral information into the audio/video stream, and 
displaying the audio/video stream to the meeting participants. 

30. (Amended) A method as in claim [29] 28, and further comprising a step of 
archiving at least the collateral information. 
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3 1 . (Amended) A method as in claim [25] 24, wherein the database queries are 
automatically generated based on information corresponding to identified topics extracted 
from the audio/video stream, where the topics correspond to predetermined topic 
taxonomies. 

32. (Amended) A method as in claim [25] 24, wherein the step of examining includes 
steps of generating a speech transcript comprised of words from at least the audio portion 
of the [audio/visual] audio/video stream; segmenting the words into sentences; and 
operating on the sentences to identify topics that correspond to predetermined topic 
taxonomies, wherein the step of generating database queries operates on identified topics. 
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